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ABSTRACT 

The NASA Johnson Space Center (JSC)/LmCom Corporation, the University of Texas Medical Branch at 
Galveston (UTMB), and the Galveston Independent School District (GISD) have teamed up to develop a Virtual 
Visual Environment Display (VIVED) that provides a unique educational experience using Virtual Reality (VR) 
technologies. The VIVED end product will be a self-contained educational experience allowing students a new 
method of learning as they interact with the subject matter through VR. This type of interface is intuitive and 
utilizes spatial and psychomotor abilities which are now constrained or reduced by the current 2 dimensional (2D) 
terminals and keyboards (2). The perpetual challenge to educators remains the identification and development of 
methodologies which conform the learners abilities and preferences. The unique aspects of VR provide us with an 
opportunity to explore a new educational experience. 

Endowing medical students with an understanding of the human body poses some difficult challenges. One of the 
most difficult is to convey the three dimensional (3D) nature of anatomical structures. The ideal environment for 
addressing this problem would be one that allows students to become small enough to enter the body and travel 
through it - much like a person walks through a building. By using VR technology, this effect can be achieved; 
when VR is combined with multi-media technologies, the effect can be spectacular. 


INTRODUCTION 

Medical students, interns, residents and physicians spend a significant amount of their time identifying and 
organizing data and information based on their educational programs or interactions with patients. It is important 
that they effectively organize this data and identify any issues that they need to learn about. Many of these 
"learning issues" can be accessed through the computer. The use of computers to process medical information, 
known as "Medical Informatics," is becoming more frequently used. This technology has advanced to 3D 
renderings of human anatomy on a 2D computer screen. Even with this technology and other classic methods, such 
as dissection of cadavers, there are times when it would be helpful for the user to be able to place himself into the 
anatomical environment to better understand the true 3D relationships that exist in that particular region of the 
body. This leads to the need for VR experiences in anatomical imaging. Studying images on a 2D computer 
screen can be compared to looking at fish through a glass bottom boat (1,3), whereas VR allows one to put on the 
scuba equipment and enter the water, interacting with the surroundings without getting wet. This type of learning 
experience would be an excellent addition to the currently available methods. 

With inevitable improvements in imaging for VR and in digital gloves, it is not difficult to visualize future 
applications for this technology. The practice of general medicine can be compared to activities of an airline pilot, 
whereby each deals with many routine landings and takeoffs, both also have to be prepared for emergency 
situations. We have the means to simulate the environment for the pilot allowing him to develop specific skills in 
routine and emergency situations. Imagine a physician putting on a VR helmet and digital gloves for a clinical 
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simulation. When the computer is activated, the physician sees the operating room, equipment, staff and patient. 
He/she holds out a hand, feels a scalpel being placed in it and begins a surgical procedure allowing him to practice 
the procedure and evaluate any possible problems before opening up the patient. 

Fidelity of the images produced is of utmost importance when applied to medical education. Magnetic Resonance 
Imaging (MRI), Computerized Axial Tomography (CT) scans and <1 millimeter cadaver sections all provide 
acceptable levels of image quality. The National Library of Medicine is using these methods of data acquisition to 
build a nationally accessible database of information on human anatomy. This project, know as the Visible Human 
Project, will provide an invaluable source of data for VR work in medical education. 

NASA JSC has used VR to expand its training capabilities. Specifically, by providing a means to incorporate 
extravehicular activities (EVA) with remote manipulator system (RMS) task training. This same VR technology is 
being transferred by way of the VTVED project for use in medical education. This is the subject of this paper. 


BA C KGROUND 

In 1990, Dr. Andrew F. Payer was invited to experience a VR prototype for astronaut training that utilized wire 
frame models of the shuttle and space station. He was placed in the VR module and allowed to translate around 
objects under his own control. After the experience, he was asked if he thought there were any possible 
applications to medical education. This was the beginning of an exciting project called the Virtual Visual 
Environment Display (VTVED) project. 

Discussions with potential commercialization partners identified the need for the project to have applications 
broader than just the medical school environment (there are only 137 medical schools in the US). It was decided 
that a joint plan be identified that would allow the VR prototypes to be tested for effectiveness in educational 
arenas including elementary schools high schools colleges and professional schools. Thus a Memorandum of 
Understanding was developed between NASA JSC, UTMB, and GISD so that prototypes could be tested in a 
variety of educational environments. 

The human skull was chosen as an initial specimen due to readily available methodology, namely CT scans, of 
obtaining imaging data for bony structures. A second model of a heart specimen was created using MRI data. 

Early models generated for VR did not meet the quality requirements of the project. After using newer CT 
scanners of higher resolution, and improvements in VR software, the skull image now nears the quality of an actual 
human skull. Because of the amount of image data, interactive fly-throughs into the skull are not currently 
possible. However, prescribed fly-throughs are being generated in digital and video formats to utilize mutlimedia 
technology. 

The short term goal of VTVED is to integrate prescribed fly-throughs of the skull with other interactive multi-media 
(audio, video, etc.) capabilities. 


METHODS: CREATING STEREO SEQUENCES OF THE HUMAN SKULL 


File Conversion and Data Preparation 

Scans of a human skull were performed resulting in a data set consisting of over 120 slices through the skull and 60 
slices through the mandible (jaw). The first attempts used 2.5 ram slice thickness and a low resolution scanner 
resulting in poor image quality. The last scan used a newer high resolution scanner and a slice thickness of 1 .5 mm 
improving the quality tremendously (see Figures 1 and 2). 

The data files obtained from the scan were transferred to JSC IGOAL (Integrated Graphics, Operations, and 
Analysis Laboratory) for further processing. The skull was held in place during the CT scan by a foam bajid thus 
creating extraneous data. The scans were then cropped to eliminate as much extraneous data as possible without 
losing any critical information. The final stage of data file preparation required using a tool developed in the 
IGOAL called "ctimager" to remove unwanted noise and extraneous data from each slice. 
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The data files obtained from the scan were transferred to 
JSC IGOAL (Integrated Graphics, Operations, and 
Analysis Laboratory) for further processing. The skull was 
held in place during the CT scan by a foam band thus 
creating extraneous data. The scans were then cropped to 
eliminate as much extraneous data as possible without 
losing any critical information. The final stage of data file 
preparation required using a tool developed in the IGOAL 
called "ctimager" to remove unwanted noise and 
extraneous data from each slice. 



Figure 1 First Skull attempts using CT data 



Figure 2 Current 3D Skull Model 


A MRI scan of a human heart was performed resulting in a data set consisting of 200 slices. In order to use the 
image tools developed to process the skull data, the data for the heart was converted to the same format as the skull 
data. Ctimager was then modified to automatically remove the noise from each scan. 


Data Filtering And Conversion Of Volume Data T o Polygonal Data 

A tool called "dispfly" was developed in the IGOAL to convert volume data into a form that could be displayed 
directly by the computer. Dispfly used multiple filtering algorithms to prepare the CT and MRI data for conversion 
to polygonal form. Anatomical models are generated based on the marching cubes algorithm developed by W. E. 
Lorensen and H. E. Cline. This technique generates surfaces based on the density of the imaging data. It generates 
a polygonal surface based on a "logical " cube created from eight data points on two adjacent slices of data. The 
algorithm determines how the surface of a selected density intersects this cube. A table lookup is performed to 
determine which triangles describe the surface. The normals are then determined using gradients computed 
directly from volume data. 

The raw data set contains noise from the CT or MRI scanning process, therefore data is filtered before generating 
the polygonal model. The filtering process typically consists of thresholding the data to eliminate most of the 
noise. A low pass filter is used to minimize the high-frequency noise that would produce an irregular bumpy 
surface when input to the algorithm. This filtering process produces a relatively smooth surface that approximates 
the scanned specimen and reduces the number of polygons generated due to noise. A unique filter was created for 
the heart data which only smoothed the data between scans, no other filtering was needed (see Figure 3). Due to the 
large number of slices in both heart and skull data sets, several models were made, each of which represented a 
small number of slices. To improve display performance a meshing algorithm was developed (meshit), that 
converted the raw collection of triangles into efficient strips. The triangle strips averaged over 100 triangles in size. 
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Generating Stereo Imag es 


After the models were made, stereo sequences were 
rendered. To generate the sequences, a tool developed by 
IGOAL called OOM (Object Orientation Manipulator) was 
used. OOM (available through COSMIC) rendered each 
frame to disk. The images used red and blue color 
separation for representing stereo images . Once the 
sequence was recorded to disk it was converted frame by 
frame to Macintosh H .pict" format and transferred to a Mac 
(full color image sequences were also transferred to the 
Mac for non-stereo viewing). 


Macintosh - Stereo Images and Multi-Media 

The Macintosh was selected because it is a relatively 
affordable platform and has a wide base already installed in 
the school systems. It is a leading engine of desktop multi- 
media and has a wide array of software and hardware 
available for this task, though more is needed. 

Once on the Macintosh, the images were edited to produce 
the desired effect, such as digitized cadaver overlays or 
text inserts describing what is being viewed. By using 
Apple's QuickTime extension, the images were then 
converted into QuickTime movies for animation on the Mac. The movies can also be edited on the Mac. To be 
able to view the stereo effects, the user must wear red/blue 3D glasses. 



Figure 3 3D Heart Model from MRI data 


A HyperCard interface (a simple tool for creating prototypes) is being created to house the Educational Experience 
on the Mac. Other hypertext programs have been investigated and will eventually replace the HyperCard interface. 
Many forms of multi-media will be considered and then incorporated into the finished product including, but not 
limited to, a Virtual Reality head mounted display (HMD) or boom system, CD ROM, laser disc, audio, video, 
digital imagery, and 3D. 


The VIVED application will contain lectures, supplemental text, graphics, digital movies, notes, pop quizzes, 
references, exams, Virtual Reality experiences of anatomy, student comments, bookmarks, qualitative analysis of 
student performance, summations, and program evaluations. 


CONCLUSIONS 

The VIVED project is at a point where CT scanned medical images of bone (e.g., the skull) can be generated into 
high quality VR imaging for prescribed fly-throughs on the Macintosh computer using a HMD or boom system. 
The project team is in the process of working on a heart VR model that has been generated from MRI data. 
Preliminary results indicate that a high resolution model can be developed using this type of imaging data. The 
project has been able to maintain its goal of high quality of VR imaging. This has led to some problems because of 
the amount of data the computer needs to deal with even during frame-by-frame sequencing of the prescribed fly- 
throughs. Alternative hardware and software options are being explored to deal with this problem. 

Another problem has been the current status of technology for the display systems for HMDs. The LCD displays 
do not have the resolution needed to maintain a high quality VR experience. The CRT displays are reaching the 
resolution needed for the project, but the cost is prohibitive for multiple education platforms. 
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The next goal for the project include improving the software and hardware for generating VR images, developing 
prescribed fly-throughs and incorporating multi-media into the VR fly-throughs. Other anatomical imaging data 
will be obtained from CT scans, MRI and Cadavers to develop VR imaging of anatomical regions that contain 
different tissues with different data densities 
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Abstract 

The technology developed for operator-in-the-loop simulation in space teleoperation has been applied to the 
Caterpillar s backhoe and wheel loader, and off-highway truck. On a SGI workstation, the simulation integrates 
computer modeling of kinematics and dynamics, real-time computation and visualization, and an interface with the 
operator through the operator's console. The console is interfaced with the workstation through an IBM-PC in 
which the operator's commands were digitized and sent through a RS 232 serial port The simulation gave visual 
feedback adequate for the operator in the loop, with the camera's field of vision projected on a large screen in multiple 
view windows. The view control can emulate either stationary or moving cameras. This simulator created an 
innovative engineering design environment by integrating computer software and hardware with the human operator's 
interactions. The backhoe simulation has been adopted by Caterpillar in building a virtual reality tool for backhoe 
design. 

Key Words: Teleoperation, redundant manipulator, simulation, recursive dynamics 
1. Introduction 

Teleoperation [ 1 ,2] involves a human operator, a hand controller, and a manipulator, in which the manipulator under 
the operator's supervision completes tasks ranging from simple trajectory following to pick-and-put operation. The 
operators interaction with the manipulator becomes a key issue, because of the intrinsic difference between a human 
operator and a robotic manipulator; that is, a human operator works more efficiently in Cartesian space [3] whereas 
most manipulators are designed with joint servo control. 

For off-line trajectory analysis, the inverse position and orientation problem can be solved iteratively using inverse 
velocity/angular velocity solutions [4], In teleoperation, however, the desired configuration of the end-effector is 
known only after the operator has commanded through the mini-master. This means that both Jacobian construction 
and inverse kinematic analysis must be completed on-line in real time. Unlike the Jacobian defined for the pre- 
determined end-effector’s trajectory, the constraint Jacobian can be constructed on-line as the operator controls the 
manipulator's end-effector through the mini-master. 

The constraint Jacobian is derived from the six constraints [5] that are imposed between the current and the desired 
end-effector's Cartesian position and orientation. The desired Cartesian position and orientation of the end-effector is 
viewed as the target that the current position and orientation will eventually have to assume. Constraining the two 
sets of position and orientation yields six constraints from which the constraint Jacobian is constructed [6]. This 
Jacobian and its pseudoinverse are used in iterations to yield the joint command angles necessary for the joint 
controllers. The procedure is illustrated with a Kraft 6-d.o.f. mini-master and a 7-d.o.f. redundant manipulator [7] 
and also with an operator's console and a backhoe. 

Since the manipulator's dynamics involves kinematic redundancy and intermittent kinematic loop closure, the 
dynamic model is constructed in the recursive Newton-Euler dynamic formulation adapted for high-speed simulation 
of general constrained mechanisms [8,9,10,1 1,12]. According to d'Alembert's principle, this formulation starts out 
with the virtual work of all the links and the cut joint, where the cut joint is expressed in the Lagrange multiplier 
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